Discriminative Learning Using Boosted Generative Models

Discriminative learning, or learning for classification, is a common learning task that has been addressed in a variety of frameworks. One approach is to design a complex classifier, such as a support vector machine, that explicitly minimizes classification error. Alternatively, an ensemble of weak classifiers can be trained using boosting [4]. However, in some situations it may be desirable to use a generative model, such as a Bayesian network, for classification. One option in this case is to train the generative model discriminatively. However, discriminative training of generative models can be computationally demanding [1, 3, 2]. In contrast, maximum likelihood learning in a generative framework can often be done efficiently, but the classification performance is frequently undermined by the need to make strong assumptions about the structure of the model and the independence of the features. In this work we propose a new framework for discriminative training of generative models. Similar to a standard boosting approach, we recursively form an ensemble of classifiers. However in contrast to situations where the weak classifiers are trained discriminantly, the “weak classifiers” in our method are trained generatively, to maximize the likelihood of the weighted data. This approach has two benefits. First, our classifiers are constructed from generative models. This is important in many practical cases when generative models, such as Bayesian networks or HMM, are desired or appropriate (e.g., sequence modeling). Second, the ML training of generative models is often much more efficient than discriminative training. The combination of discriminative weighting of the data with generative training of the intermediate models yields a computationally efficient method for training generative classifiers. We introduce a new discriminative structure learning method, called Boosted Augmented Naive Bayes (BAN) classifier. We demonstrate that BAN is easy to implement and computationally efficient. BAN’s performance on a large suite of benchmark datasets is superior to naive Bayes, TAN, and generatively-trained Bayesian networks. It is competitive with BNC-MDL, BNC-2P, ELR-NB, and ELR-TAN and requires significantly less computation during training. We also demonstrate the benefit of parameter boosting in discriminative training of dynamic Bayesian network classifiers. Our initial results show that boosted DBNs always outperform models which are trained using standard maximum likelihood methods. The computational complexity of our weighted maximum likelihood approach is within a constant factor of standard maximum likelihood learning, making it ideal for discriminative training of complex models over large datasets.